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ABSTRACT 

Research at the United States Army Aviation Schopl^ 
Fort Rucker, Alabama^ sought to improve the school-wide training 
quality control system. Investigators studied: 1) the relation 
between the grades a student received from instructors and those he 
received from a checkpilot; 2) the effect of checkpilots' prior 
information about students • progress upon the grades they assigned; 
3) inter-rater differences in grading practices; and 4) the 
usefulness of checkpilot comments on grade slips. Performance records 
from student classes of 1961 and 1962, when grading chapges were 
introduced, were examined. Analysis showed substantial correlations 
between instructor and checkpilot evaluations when checkpilots were 
drawn from the instructor pool, but not when they came from putside 
sources. . The individual standards of the rater influenced final 
evaluations, and optimum procedures were not used in the collection 
of student performance data. These findings suggested that the flight 
grading system needed standardization^ that an organizational 
separation of training and assessment functions was desirable, and 
that the data input for quality control should be made more 
objective, detailed, and routinized. (PB) 
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FOREWORD 



This report presents an analysis of grading practices and procedures in 
effect in rotary wing training at the U.S. Army Aviation School at Fort Rucker, 
Alabama, during 1961-G3 instruction periods. The study was requested by the 
School's Rotary Wing Department in connection with continuing development of 
quality control procedures in the training program. 

A major purpose of this analysis was to compare the results of training and 
checkflight procedures in effect at that time with the results from a HumRRO 
study conducted in 1956-57 under Work Unit LIFT. The LIFT study analyzed 
training and checkflight grades for Army helicopter students at the Aviation 
School and at the U.S. Army Primary Helicopter School at Fort Wolters, Texas. 

Another objective in the present analysis was to use the experience and 
results as a basis, if appropriate, for generalizations about characteristics 
needed in data to be used in training quality control systems and procedures in 
a broader context. 

Both this studyand the earlier LIFT research were performed by HumRRO 
Division No. 6 (Aviation) at Fort Rucker. The present study was undertaken 
as Technical Advisory Service, and initial results were reported to the Aviation 
School. This report was developed to make the information more generally 
available to those interested in the development and operation of quality control 
systems for training. 

The present analysis was begun by Dr. Wallace W. Prophet, with Dr. J. 
Daniel Lyons as Director of Research. It was completed by Dr. Paul W. Caro, Jr., 
with Dr. Prophet as Director of Research. 

Military support for the research was supplied by the U.S. Army Aviation 
Human Research Unit. MAJ Donald J. Haid was the Unit Chief at the time the 
research was conducted. 

The grading practices analysis conducted under Work Unit LIFT (Army 
Aviation Helicopter Pilot Training) was reported in HumRRO Technical Report 77, 
Improving Flight Proficiency Evaluation in Army Helicopter Pilot Training , by 
George D. Greer, Jr., Wayne D. Smith, and CPT Jimmy L. Hatfield, May 1962. 
Subsequent work under Work Unit LIFT is described in a Consulting Report, 
A System of Flight Training Quality Control and Its Application to Helicopter 
Training, by John O. Duffy and Carroll M. Colgan, June 1963. 

HumRRO research for the Department of the Army is conducted under 
Contract DA 44-188-ARO-2 and Army Project 2J024701A712 01, Training, 
Motivation, Leadership Research. 



Meredith P. Crawford 
Director 

Human Resources Research Office 



SUMMARY AND CONCLUSIONS 



Military Problem 

A training system-like a factory producing an item for market-can benefit greatly from 
quality assurance procedures and techniques integrated into a formal quality control system. 
Such a system can assure that graduates of the training system are able to function at required 
levels of proficiency, and it can provide managers with diagnostic data for continuing improve- 
ment of the training. 

The U.S. Army Aviation School, port Rucker, Alabama, has had strong interest in establish- 
ing a schooi-wide flight training quality control system. As a step toward this end, the School 
requested that HumRRO study several aspects of the procedures used in evaluating the training 
and performance of the School's rotary wing trainees. 

A study conducted in 1956-57 by HumRRO Division No. 6 (Aviation) had shown that there 
was little relationship between the grades assigned to Army helicopter students by flight instruc- 
tors and those assigned by checkpilots, and that end-of -course evaluations were greatly affected 
by the individual standards of the checkpilots. Since that time, a number of changes in evalua- 
tion procedures had been instituted. The study described in this report was conducted in 1963 to 
obtain information on the current characteristics of flight grading practices of the Rotary Wing 
Department of the Aviation School. 

Research Problem 

A quality control system for training includes the following elements: 

(1) Detailed specification of training goals 

(2) Accurate and appropriate proficiency evaluations 

(3) Effective communication concerning training efficiency 

(4) Effective procedures for corrective action 

(5) Supervisory support 

If quality control evaluation procedures based on data from examiners are to be effective, 
the examiners must use common standards of evaluation and must provide detailed, reliable, and 
valid information inputs for the system. In viev/ of these general quality control requirements, 
the present study was directed toward four main questions about flight grading practices: 

(1) What is the relationship between grades given by instructors and checkride grades 
given by checkpilots? 

(2) What ore the effects on checkride evaluations of checkpilots' prior information 
about students? 

(3) What variations exist in individual checkpilot standards and grading practices? 

(4) What is the usefulness, for quality control purposes, of checkpilot comments on 
grade slips? 

A corollary objective was to seek findings of broad applicability relevant to techniques 
and procedures for training quality control systems in general. 

Procedures 

This study was based on the performance records of students in selected rotary wing 
classes spanning the 1961''63 period. 



Flight records for six FY 1961 and seven FY 1962 classes were analyzed to determine the 
extent of similarity between grades assigned by instructors and those assigned by checkpilots. 
These two samples of classes were selected to represent "before and after" situations related 
to changes introduced in the School's grading practices in FY 1962- 

Study was also made of whether grades were influenced by the checkpilots' relationship 
to, or possible familiarity with, the students or their instructors. Two ^Y 1963 classes, with a 
combined enrollment of 60 students, were selected for this experiment. At that time it was the 
practice in the Rotary Wing Department for instructors within an instructing flight to administer 
checkrides to each other's students and one-third of the 60 students in the experiment were 
evaluated according to this procedure. The remaining two-thirds were given checkrides by check- 
pilots drawn from outside the classes studied, a procedure intended to obviate bias in checkpilot 
grades that might result from prior knowledge of the student's performance. 

A third aspect of the research concerned the extent to which the individual checkpilot's 
own standards might influence the grades assigned to students. The seven 1962 rotary wirig 
training classes were used as the basis for this analysis. In these seven classes, there were 
17 checkpilots who had given 12 or more checkrides. The grades assigned by these checkpilots 
were analyzed in detail. The records^of the checkpilots in the two 1963 classes used in the 
preceding experiment v/ere also analyzed. 

A fourth consideration was whether the kind of diagnostic information appropriate for quality 
control programs was available under the grading system used by the Rotary Wing Department. To 
ascertain this, comments made by checkpilots on student grade slips v/ere reviewed and classified 
according to level of specificity. 



Results 

(1) Substantial correlations-which conflicted with the low correlations reported in the 
1956-57 findings-were found between instructor and checkpilot evaluations of student perform- 
ance, with checkpilots selected from among the instructors of the students' own instructing flights. 

(2) There was little relationship between the evaluations of instructors and those of check- 
pilots when checkpilots were drawn from outside the students' flights. 

(3) End-of-stage evaluations of student performance were significantly affected by: 

(a) The individual standards and grading practices of the checkpilot. 

(b) Whether the checkpilot was a member of the student's own flight. 

(c) The stage of training at which flight performance was being evaluated. 

(4) Some of the specific information on individual student performance needed for an effect- 
ive quality control program was recorded by checkpilots under the existing grading system, but 
it was not collected in a consistent manner or processed for systematic diagnostic use. 



Conclusions and Implications 

(1) The insufficient correlation between instructor and checkpilot grades, and the large 
variations in individual checkpilots' standards and practices, suggest the flight grading system 
falls short of what is needed for on effective school-wide quality control system. 

(2) While some information useful for quality control purposes was being collected, addi- 
tional mechanisms and procedures for assuring consistent collection, recording, summarizing, 
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Qi.d reporting of the data needed to be devised in order to increase the effective use of 
such information. 

(3) A procedure in which instructors evaluate and report upon the qualitv of the products 
(students) of their own training unit cannot easily yield information upon which to base an effec- 
tive quality control system. An optimum system requires the organizational separation of assess- 
ment and training functions, and uniform procedures for performance evaluation. 

(4) Data input for training quality control purposes must meet certain criteria. Ideally, the 
data should be based on standardized proficiency tests administered under standard conditions. 
Data should be detailed, objective descriptors of trainee performances if the feedback of informa- 
tion to the training operation is to allow effective adjustments in that operation. 
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INTRODUCTION 



Quality Control System Requirements 

For many years the concept of quality control has been an accepted feature 
of industrial management, and a formal quality control system has been a 
routine aspect of manufacturing operations. The detailed assessment of product 
quality under such a system serves two basic functions: (a) It assures that the 
products going to the market meet minimum standards of quality, and (b) it 
provides relevant feedback to management as to changes needed in the production 
process to bring more of the products to the irinimum standard, to raise the 
standard, or to lower costs* The procedures used and general organizational 
structure involved in industrial quality control have become well systematized 
and documented* 

The two general functions of quality control are just as pertinent to a 
training system as to a manufacturing system* Only recently, however, has 
there been any systematic application of the concepts of quality control to the 
production of skilled personnel through training. While proficiency tests and a 
wide range of training and administrative techniques to measure and upgrade 
the duty qualifications of trainees are part of every military training program, 
the planned use of all phases of an integrated quality control system is not yet 
common in the training environment. 

The general concept of training quality control has been discussed in detail 
by Smith (1), and a specific application of the concept in a military training con- 
text is described by Duffy and Colgan (2). They outline five general require- 
ments for an effective training quality control system: 

(1) Detailed specification of training gOcils 

(2) Accurate and appropriate proficiency evaluations 

(3) Effective communication concerning training efficiency 

(4) Eifective proce^*ures for corrective action 

(5) Supervisory support 

While there is little doubt as to the desirability of controlling quality in 
a training system, particularly in one that is directed toward producing critical 
and complex skills, the feasibility of conducting an effective training quality 
control program will vary from one organization to another as a fimction of 
many factors. . 

The rapid growth of Army av ition training, particularly in the rotary wing 
area, has intensified command int .-rest in developing procedures that will allow 
for better management of the extensive and complex training system* A train- 
ing quality control system is one such procedure* The problem of training 
quality is critical to a school such as the U*S* Army Aviation School (USAAVNS) 
at Fort Rucker, Alabama, which in recent years has been devoting more and 
more attention to this subject* That concern has led to ♦he institution of a num- 
ber of procedural changes at the School and provided the impetus for the research 
described in this report. 

One factor important to the success of a quality control system is the kind 
of data input on student performance that is available for evaluation purposes. 



In flight training, the quality control requirement for "accurate and appropriate 
proficiency evaluations" can be supplied through indices of performance such 
as daily flight grades and checkride grades assigned at the end of various stages 
of training. Performance checks serve two purposes: (a) They demonstrate 
whether the individual product (student) meets the required standards of per- 
formance (i.e., whether the training objectives have been met), and (b) most 
important for a quality control system, they provide information that serves as 
feedback to the personnel responsible for ' '^^ ring the overall instruc- 
tional system. Such feedback signals the o .^nd forms the basis for, 
changes or adjustments in the training progicxin. 

The effectiveness of a training quality control system will - therefore be, 
in large part, a function of the extent to which information on the quality of the 
product—student performance— is valid and reliable. Examination, by research 
analysis techniques, of certain aspects of the information on student flight per- 
formance that is available for quality control use at the Aviation School was 
undertaken at the SchooVs request, as a result of growing interest in establish- 
ing a formal training quality control system; . 

The data which form the principal basis for the various phases of the present 
study are from Aviation School classes during FY 1961-63. An overview of the 
grading practices and quality control procedures in effect at the School is pre- 
sented in the following section. Certain developments in flight grading and 
quality control that have taken place at USAAVNS during FY 1964-66 are 
described in the final section of thi^ report. 

Flight Grading and Quality Control at USAAVNS 

Flight evaluation practices and procedures at the Aviation School differ 
from those of the U.S. Army Primary Helicopter School at Fort Wolters, Texas, 
where a formal quality control system is in operation, principally in terms of 
the type of data input to the system. 

The USAPHS system (described by Duffy and Colgan, 2, and Duffy and 
Anderson, 3), utilizes a detailed, relatively objective checklist of flight perforni- 
ance items. Data are in the form of deviations from desired values in airspeed, 
altitude, RPM, and so forth. Such data input can be characterized as objective 
descriptors of performance . The objective checklist is known as the Pilot Per- 
formance Description Record (PPDR). Development of the PPDR is described 
by Greer et al . (4). 

The data input available for quality control use at USAAVNS can be described 
as a subjective evaluation of performance . In this system (described by 
USAAVNS, 5'), the checkpilot or instructor provides a numerical or letter score 
or rating which is the evaluation of the student's performance. In addition, com- 
ments of varying degrees of specificity may be written on the grade record 
sheet by the checkpilot or instructor. 

While USAAVNS has not had a school-wide quality control system, 'for a 
number of years a rather thorough and extensive quality control system has 
been operated by the Department of Fixed Wing Training for the primary phase 

of their ino':ruction.^ That system has many features in common with the one 

^,,\ , 

^Reference 5 is the USAAVNS Regulation that was in effect at the time this study was conducted. A 
new USAAVNS Regulation 350-16, dated June 1963, governs current USAAVNS grading practices. However, 
the system embodied in the current regulation is generally like that described in 5. 

^During FY 1966, the primary fixed wing system was extended to other phases of fixed wing training, 
and efforts are under way to extend it to all USAAVNS flight training. 



at USAPPIS, and in fact provided the impetus for some of the features of the 
USAPPIS system. The principal differences relate to (a) the use of subjective 
evaluations of overall performance at USAAVNS versus the objective indices of 
details of the performance at USAPHS, (b) administration of checkrides by 
instructors from within the student's own flight at USAAVNS versus administra- 
tion by a separate checkpilot group at USAPHS, and (c) the use of hand-processing 
of data at USAAVNS versus the use of automatic data processing equipment 
at USAPHS. 

Since the present study dealt with several aspects of the quality control 
data input at USAAVNS— that is, the daily and checkride flight grades— some 
additional details on the manner in which these grades are derived will be 
presented later (see p. 6) to provide background for subsequent description of 
the research. The general characteristics of subjective flight grades have been 
discussed by Greer et al . (6), and will not be repeated here. 

Army flight training is divided into a number of phases and stages, each 
of which has certain definable objectives or goals. During the course of the 
training, each dual instructional flight— unless maneuvers are being demon- 
strated to the student for the first time— is given a summary letter grade' by 
the instructor. In addition, the instructor will assign a letter grade to the per- 
formance of specific maneuvers and to the student's basic qualities, and he may 
also write narrative comments. As the various stages are completed, the 
student's proficiency is checked to see whether he has met the objectives of 
that stage and is prepared to progress to the next stage of training. 

These checks on proficiency (or checkrides) are normally administered 
by someone other than the student's own instructor.^ In most cases, the check- 
rides are administered by "checkpilots" who are instructors in the same 
instructing flight or group as the staJent's own instructor.^ In some cases, the 
checkpilot is an instructor from another flight or is from an administrative 
office of the instructing department or school, such as the Standards Division. 

The assignment of students to particular instructors for the periodic rheck- 
rides is generally nonsystematic, in that no method is specified. In the Depart- 
ment of Rotary Wing Training the flight commander concerned usually makes 
the assignments in accord with current workloads and such other factors as he 
deems pertinent. In some cases there may be a pattern to the assignments, 
such as assigning marginal students to particular instructors whom the flight 
commander considers especially adept at checking weak students. 

Details of the checkride gilding system are described by USAAVNS (5). 
The checkride produces a single numerical grade or letter grade that summa- 
rizes the evaluation of the student's flight performance on the checkride. 
Generally, above-average performance is assigned grades in the 90s, average 
in the 80s, and below-average in the 70s; unsatisfactory performance, or failure, 
is assigned the letter grade U. In addition, performance on specific maneuvers 
is assigned letter grades A, B, C, or U, as is the checkpilot's assessment of 
the student's "basic qualities," such as motivation and coordination. Finally, 

^The exact terminology for these grades has varied from time to time, but can generally be characterized 
as versions of a four-point rating scale, e.g., /I, /i, C, and {/, or AA (above average), /i, /i/i, and U- 

^Thc most common exception is the administration of the pre-solo checkride, which mn\ be done by the 
student's own instructor. 

'An exception to this involves instruction administered by contractor personnel, i.e., non-military or 
non-Dcpartmcnt of the Army civilian instructors. Students of contract instructors are checked by designated 
military or Department of t!*e Army civilian chcckpilots. Data used in the present study did not involve 
contract instructors. 



the checkpilot may provide narrative comments on specific aspects of the 
5;tudent*s performance. 

As the student completes a stage of training, his instructor assigns him an 
"instructor evaluation" grade, which is a numerical grade utilizing the same 
scaling as the checkride grade, and which can be thought of as the instructor's 
estimate of the grade the student will receive on his end-of-stage checkride. 

In summary, at USAAVNS the following subjective evaluation grades or 
comments are available for use in quality control: 

(1) Instructor daily summary letter grade. 

(2) Instructor daily letter grade on specific maneuver performance. 

(3) Instructor daily letter grade on student's basic qualities. 

(4) Instructor narrative comments on specific aspects of the student's 
daily performance. 

(5) Instructor end-of-stage numerical evaluation grade. 

(6) Checkpilot *end- of- stage numerical (or letter) summary check- 
ride grade. 

(7) Checkpilot end-of-stage letter grade on specific maneu- 
ver performance. 

(8) Checkpilot end-of- stage letter grade on student's basic qualities. 

(9) Checkpilot narrative comments on specific aspects of the student's 
performance on the checkride. 

Rationale for Present Study 

As previously indicated, the present study stemmed from USAAVNS concern 
over quality control procedures that would be appropriate for School use. Earlier 
studies (1956-57) with Army helicopter students, reported by Greer et al . (6), 
had raised a number of questions about the reliability and validity of subjective 
evaluations of flight proficiency. It was decided to examine evaluation data from 
more recent USAAVNS classes to see whether the data characteristics had 
changed. The periods selected were FY 1961 and FY 1962, co provide informa- 
tion on evaluations before and after certain changes in USAAVNS grading, the 
most notable being the introduction of the concept of "basic qualities." 

The basic concern in t^e analyses of grading information for the two time 
periods covered was with che correlations between instructor-asuigned daily 
flight grades and checkpilot-assigned checkride grades. The earlier study of 
USAAVNS grading had shown that these two indices were generally not corre- 
lated with one another. This lack of correlation between two independent meas- 
ures of the capabilities of the same students was interpreted as indicating 
insufficient reliability and validity in the flight grading system. Considerable 
variation was found in the standards of evaluation used by individual checkpilots 
in that study, so this matter was also treated in the present study. 

Another point of concern which was explored in the present study stemmed 
from the administrative organization underlying the assignment of personnel to 
administer checkrides at USAAVNS. In their analyses of quality control systems. 
Smith (t) and Duffy and Colgan (2) emphasize the importance of having the 
quality control element of the organization distinct and separate from the instruct- 
ing element, both as to administrative organization and as to personnel. Such 
a procedure— which would, in essence, require a separate group of specialists 
who did nothing but administer quality control checkrides— has not been con- 
sidered feasible by USAAVNS because of the organizational and personnel prob- 
lems arising from the many different flight training courses offered at the 
School and the number of different aircraft types involved. Under the present 
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organization of USAAVNS, therefore, it is not possible for all checkrides to be 
administered by persons outside the instructing flights. 

To obtain information on how this procedure may affect flight evaluations, 
data from checkrides under standard USAAVNS practice were compared with 
results of checkrides administered by personnel who were not members of the 
student*s own instructing flight. These special checkride data were gathered on 
certain FY 1963 classes as part of the research activities. 

The organizational problems that have prevented USAAVNS from having 
separate, independent groups of checkpilots also make it difficult to utilize 
objective performance measures such as the PPDR. Use of the PPDR is a 
specialized checkpilot skill that requires special training, and a checkpilot needs 
to administer the PPDR regularly if he is to maintain his level of skill in its 
administration. These factors make it necessary for the checkpilots to be a 
separate group whose specialty and duty is the administration of checkrides.. 
The less variecJ flight training mission of USAPHS permits the organization 
necessary to routine use of the PPDR in the quality control system at Fort 
Wolters, whereas the much more varied flight training mission of USAAVNS 
would pose considerably greater problems for efficient operation with such 
an organization. 

Previous HumRRO research (6), as well as that of other researchers (7), 
has noted the sometimes considerable variability among checkpilots in the 
standards they use in evaluating student flight performance. It is common 
practice for students to characterize various checkpilots as "Santa Clauses" 
and "Hardnoses." Therefore, in another phase of the present research, analyses 
of checkpilot records from the seven FY 1962 classes were made to determine 
whether there were significant differences in checkpilot standards reflected in 
these data. 

A final area of concern in the present study was the usability of the narra- 
tive comments on USAAVNS grade slips for quaTliy control purposes. The 
necessity, for quality control, of having detailed and specific information about 
student performance has been stressed by Smith (1). Since the comments on 
the grade slips provide the most detailed and specific information in the USAAVNS 
grading sy.3tem, these comments were examined for the FY 1963 classes studied. 

ANALYSIS OF TRAINING AND CHECKRIDE GRADES 

Problem and Approach 

The purpose of the first phase of the research was to determine the extent 
to which the grades assigned to students by instructors agreed with those assigned 
by checkpilots for the same stage of training. Coefficients of correlation were 
computed between the instructor's evaluation grade (IE) assigned the student 
just before an end-of-stage checkride and the checkpilot*s grade (CK) assigned 
the student on the basis of the end-of-stage checkride. If the instructor and the 
checkpilot are applying similar standards of performance in their respective 
evaluations of the student, a substantial correlation^ will exist between the two 
sets of evaluations (although substantial lE-CK correlations can exist for rea- 
sons other than commonality of standards). 

'The magnitude of such correlations will not nppronch the test-retest relinbilitics usually associated 
with psychometric tests. Kor further discussion of this point see Prophet (8). 
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Near-zero relationships between the two grades had been reported in 
1956-57 (6). The flight grading system under which the low lE-CK correlations 
were found had undergone changes, as described in a 1961 USAAVNS flight grad- 
ing memorandum (5) instituting a concept of evaluation of flight performance 
"basic qualities". To determine whether there had also been a change in the 
lE-CK grade relationships, data from two separate groups of rotary wing classes, 
representing the "before" and "after" situations with regard to the flight grading 
system changes, were utilized in the correlational analysis. 

Procedure 

The flight records for six FY 1961 and seven 1962 classes were selected for 
study. These classes represented both the Officer Rotary Wing Aviator Course 
and the Warrant Officer Rotary Wing Aviator Course given at the Aviation School, 
These 13 classes were selected for the analysis because (a) complete records 
of student performance were available and (b) these classes represented stable 
grading practices during periods before and after the changes in grading prac- 
tices initiated in mid-1961. The classes selected are listed in Table 1. 



Table 1 

Classes Selected for Analysis 
of Grading Practices 



Before Chnnjict. in 


After Changes in 


Grndin^ 


Practices 


Grading pMcticcs 


Class 


Number of 


Class 


Number of 


Number 


Stu(}ent^ 


Number^ 


Students 


61-1 


36 


62-l\V 


38 


61-2 


30 


62-2\V 


42 


61-3 


28 


,62-3 


35 


61-4 


29 


62-5 


'k) 


61-5 


19 


62-6 


28 


61-6 


15 


62-7 


27 






62-8 


43 


Total 


157 


Total 


259 



"(Mass 62-4 was canceled. 



The grade folder of each student in the classes selected was examined, and 
the daily grades (DC), instructor evaluations (IE), and checkride grades (CK) 
for each stage of training_were abstracted for use in the correlational analysis. 
The mean daily grade (DC) for each student for each stage of training was com- 
puted. Pearson product-moment coefficients of correlation were then computed 
among these three variables (DG, IE, and CK) for each stage of training. These 
correlations are shown in Table 2. 

Results 

As previously stated, the correlations of principal interest are those between 
instructor evaluations and checkride grades. The lE-CK correlations for the 
six FY 1961 classes combined (the "before" group) and for the seven FY 1962 
classes combined (the "after" group) are presented for each stage of training 
in Table 3. Also shown is the range in correlations for individual classes in 
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Toble 2 

Correlations Between Mean Daily Grades, Instructor Evaluations, 
and Checkride Grades for Each Stage of Training *i 



rr.ninri^ 


Viirijhlc 






|n*«lriinieiii 
(!ro>»>^CoHniry 


no IK CK 


no IK CK 


DC IK 1 CK 


Prc-Soln 


D.n'ly (fM(li: 

Instructor 
l^vnluation 

Clicckridc 


.70 .21 

.64 - .39 
.26 .47 


.24 .2.1 




Adviinucd 


Daily Grade 

Instructor 
Fvaluntion 

Chcckridc 


.38 
.24 


.67 .45 

.67 - .55 
.36 .41 




ItiHtrumcnt 

Cross-Ciountry 


Daily Grade 

Instructor 
Evaluation 

Chcckridc 


.19 


.27 
.22 


- b b 

.5t - h 

.23 .48 



♦'Corfficicnls in ihc upper rijthi hand portion of ihc I'rc-Solo and ihc Ads-anc-H Mapc analyses vvrrc com- 
puted from data from six KV 1961 classrs (A'=-l.->7> iraincd bc-fom ihc changns in flighl grading practices in 1901. 
Coefficicnl.s in ihe lovser left hand porlions were compult-d from daia from .seven KV 1962 classes (.Y ^259) trained 
after these chnugcs. 

^'The Instrtinient Cross-Country phase wns not included in tho training program during V\ 196 1. 



Table 3 



Correlations Between Instructor and Checkpilot Grades 
Before and After Grading Practices Changes 



Training 
Stnge 


Before Changes in 
Grading Practices 

(A:=IS7) 


After Changes in 
Gradirrg Practices 
(A' = 259) 


Total Group 
Correlation 


Inter-Class 
Hango 


Total Group 
Correlation 


Inter-ClaKs 
Range 


I'fc-Solo 


.39 


.16- ,65 


.47 


.34- .63 


Advanced 


.55 


.39- .81 


.41 


.21 -.59 


Instrument 










Cross-Country 


a 


a 


.48 


.01-. 73 



"Not applicable. This stage was not instituted until FV 1962. 



the "before" and "after" samples. Means and standard deviations for the DC, 
IE, and CK grades for each class are given in Appendix A, 

Each of the correlations reported for the total group in Table 3 is statis- 
tically significant (£<.001). These correlations are similar in magnitude to 
relationships reported elsewhere in studies of flight training research (7). They 
indicate substantial agreement between instructors and checkpilots in the 
assignment of grades in the Department of Rotary Wing Training, both before 
and after the changes in grading practices. None of the differences between 
"before" and "after" mean correlations was statistically significant. 
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Discu ssion 

On the basis of these data, the near-zero correlations observed in rotary 
wing training in 1956-57 did not obtain in the Department of Rotary Wing Train- 
ing, USAAV^S, during the period covered by the analyses presented in the 
present study. On the contrary, substantial relationships did exist between 
instructor evaluations and checkride grades for all three stages of advanced 
rotary wing training. These relationships were not significantly affected by 
grading practice changes introduced by USAAVNS in mid-1961. 

However, it is possible for such substantial relationships to exist on bases 
other than commonality of flight evaluation standards between instructor and 
checkpilot personnel. The existence of substantial correlation between IE and 
CK is a necessary condition for a conclusion that instructor and checkpilot 
grades are based on the same standards for evaluating student performance. 
It is not, however, in itself enough to justify such a conclusion, since the com- 
mon frame of reference for instructor and checkpilot could have been based on 
something other than the student's performance. This aspect of the evaluation 
was explored in the second phase of the research. 

THE EFFECT OF PRIOR KNOWLEDGE ON 
CHECKPILOT EVALUATIONS 

Problem and Approach 

The relationship between instructor evaluations and checkride grades can 
be significantly influenced by a number of factors^ other than the standards 
applied by instructors and checkpilots. One possible biasing factor is the 
amount of information the checkpilot has about the student prior to the checkride. 
Such information may produce spuriously high correlations. 

Ideally, if the checkride grade is to be an unbiased evaluation of the student's 
performance, the checkpilot should have no prior knowledge of the student's 
performance capabilities. In practice, it is virtually impossible to attain this 
ideal in a situation where the checkpilot is an instructor who has been in daily 
contact with the students and with their instructors. 

At the time of the research reported here, a separate check section did not 
exist in the Department of Rotary Wing Training, USAAVNS. It was the practice 
for the instructors to administer checkrides to each other's studentr,. The 
second phase of this study sought to determine whether the organizational 
assignment or location of the checkpilot— that is, whether he was an instructor 
within the student's own flight or from outside that flight— influenced the rela- 
-tionship between grades assigned by instructors and those assigned by check- 
pilots. It is assumed that if such influences do operate, the probable reason 
would be the information available to the checkpilot, prior to the checkride, 
concerning the particular student or his instructor. Therefore, a number of 
checkrides were administered under controlled conditions intended to reduce 
the amount of prior information to which the checkpilots were exposed. 

Procedure 

The classes selected for this experiment were Classes 63-1 W and 63-3. 
The combined enrollment of these two classes was 60 students. 

*Maiiy of the factorH that can biof* checkride evaluations are discusHed in the PPDR llandhook (4). 
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Five pilots were selected by the Department of Rotary Wing Training to 
administer "Special" checkrides. These checkpilots, who were judged competent 
by USAAVNS to make evaluations of student performance, had participated 
in frequent and regular evaluations of students comparable to those in the 
classes under study, and had no duties that required regular contact with stu- 
dents or instructors in either of the classes in the experiment. Four were 
instructors from other classes, and the fifth, who served as an alternate, was 
from the Standards Section of the Department of Rotary Wing Training. 

Since these five checkpilots were not instructing in the flights under study, 
it was assumed they would have little or no exposure to the kinds of information 
about the performance of students they would be evaluating that would be apt to 
bias their evaluations. The selection of these five checkpilots did not assure 
unbiased evaluations of student performance; the only intent was to minimize 
the amount of information about the students that would be likely to be available 
to the checkpilots prior to administering the checkrides. 

Approximately two-thirds of the 60 students in the selected classes were 
given checkrides by the Special checkpilots. Tht remaining students were eval- 
uated by regular pilots in accordance with the procedures then in effect in the 
Department of Rotary Wing Training. 

To further reduce the amount of information that might be available to 
Special checkpilots prior to the administration of checkrides, the research 
staff selected the students for special checkrides and assigned the Special 
checkpilots to evaluate them. This was done, using a random selection pro- 
cedure, after the flight commander had designated the students who were to be 
given checkrides on a specific date. The individuals concerned were notified 
immediately prior to the checkride. Under this procedure there was little 
opportunity for the Special checkpilot to be exposed to possibly biasing infor- 
mation about the student he was to evaluate. 

One restriction— that no Special checkpilot could administer a checkride 
to a student he had evaluated on a previous checkride— was placed upon the 
random selection procedure. Therefore, no Special checkpilot administered 
more than one checkride to a given student., The numbers of students receiving 
Special and Regular checkrides during each stage of training are presented in 
Table 4. 



Table 4 

Correlations Between Instructor and Checkpilot Evaluations" 









StHgC 0 


f 1'riiining 






cticckridc 


Pfe-Solo 


Advanced 


Instrument 
Crohs-Country 




A' 


Correlation 


A' 


Correlation 


A* 


Correlation 


Special 


36 


.28 


40 


.20 


41 


.'8 


Regular 


24 


.5r>^ 


20 


.73^ 


18 


.64 



«Clasj>c.s 63-lW and 63-3. 

^CocfficirntH significantly greater than zero (;)<.0I). 



Results 

The Pearson product-moment coefficients of correlation obtained between 
instructor evaluations and checkride grades for the Special and Regular checkrides 
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administered during this study are reported in Table 4. For purposes of these 
analyses, data from the two classes were combined. 

For all three stages of training, the checkride grades assigned by the 
Special checkpilots showed less relationship to instructor evaluations than did 
the evaluations assigned by Regular" checkpilots. Coefficients of correlation 
between end-of-stage evaluations assigned by instructors and those assigned by 
checkpilots were substantial {£< .01) for each stage of training when the check- 
rides were administered under the regular system. On the other hand, the 
corresponding correlations were not significantly different from zero when 
administered under the special conditions established for this study. 

The correlations for the regular group reported i) Table 4 are in keeping 
with the higher correlations found in the first phase of the study for similar 
comparisons for certain of the FY 1961 and FY 1962 classes;. However, under 
the independent conditions established for the second phase K this study, no 
significant agreement was found between instructor and checkpilot evaluations. 

Discussion 

Evidence from the statistical analysis, in the first phase of the present 
study, of checkride grades in the FY 1961 and FY 1962 classes indicated sub- 
stantial agreement between instructors and checkpilots in assigning grades. 
Analysis of checkride grades administered under controlled conditions in the 
second research phase did not provide indication that such agreement reflected 
a similarity of evaluation standards between the two groups of evaluators. On 
the contrary, it suggested that the apparent similarity of evaluation standards 
may have been an artifact attributable to availability of information to check- 
pilots prior to checkride performance evaluation. 

On the basis of the evidence from the experimental checkrides, it appears 
that the conclusions of the 1956-57 study with respect to grading standards still 
had applicability in 1963. The ratings of students' checkride performance still 
tended to reflect something other than the performance of the students being 
evaluated. The tiird phase of the study was designed to investigate the hypothesis 
that a major factor in the assignment of grades by checkpilots was the check- 
pilot's own standard of evaluation. 

VARIATIONS IN INDIVIDUAL CHECKPILOT STANDARDS 
AND GRADING PRACTICES 

Problem and Approach 

A lack of agreement between checkpilots and instructors with respect to 
the evaluation of end-of-stage student performance was demonstrated in the 
second phase of the present study. The 1956-57 study reported that such lack 
of agreement reflected differences in the individual standards of checkpilots (6). 
An analysis was therefore made of the evaluation records of individual check- 
pilots to determine whether they had personal standards and grading practices 
that differed significantly and that significantly influenced the grades of the 
students whom they evaluated. 

Procedure 

Data from the seven FY 1962 rotary wing training classes listed in Table 1 
were used in this phase of the study. In order to explore the possible variations 
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in grading practices among checkpilots in the seven classes, grades assigned 
by each of them were assembled. Since many of these checkpilots had admin- 
istered too few checkrides to provide a stable indication of their grading stand- 
ards, it was decided that only those checkpilots who had administered 12 or 
more checkrides would be included in the analysis. There were 17 such pilots. 

Grading data from both Special and Regular checkpilots from the two 
FY 1963 classes used in the second-phase experiment also were analyzed in 
the study of individual variations among checkpilots. 

Results 

The means and standard deviations of checkride grades assigned by the 
17 pilots from the FY 1962 classes, for each stage of training and for all stages 
combined, are reported in Table 5. The mean grades for all stages of training 
combined ranged from a low of 77.1' (Checkpilot Q) to a high of 87.9 (Check- 
pilot A). The standard deviations ranged from 3.1 (Checkpilot A) to 7.9 
(Checkpilot Q). 

As might be expected when smaller numbers of checkrides were involved, 
the means and standard deviations within the various stages showed wider 
variations than for all stages combined. The largest difference between 



Tables 

Means and Standard Deviations*' 
of Grades Assigned by Selected Checkpilots 













Stage 


of Training 










Check, 
pilot 


Prf-Solo 




Instrument 
Cro>s-Country 


All Stapes 




A' 


Mean 


SI) 






SI) 


iV 


Mean 


SI) 




Mean 


SI) 



A 
li 
C 
D 

E 
F 
G 
II 

I 
J 
K 
L 

M 

N 

0 

V 

Q 
All 
Check, 
pilots 



.*> 
10 
4 

4 
2 

7 
2 

10 
10 

8 
5 

10 
5 
0 
3 
9 



86.4 
87.4 
86.5 
86.:") 

82.8 
86.0 
84.7 
91.0 

84.4 
84.3 
83.0 
83.0 

80.2 
80.8 

72.3 
78.1 



3.4 
l.f) 
hXy 

2.9 
4.0 
5.1 
0.5 

3.4 
{.6 
4.3 
6.9 

9.4 
8.4 

7.1 
6.9 



1 
8 
19 
10 

5 
9 
9 
4 

5 
3 
8 
5 

10 

7 
9 
5 
3 



87.8 
86.6 
85.6 
86.5 

89.0 
86.1 
86.6 
84.8 

85.0 
86.7 
8k9 
80.8 

81.7 
80.3 
79.2 
79.8 
68.7 



99 83.3 6.7 123 84.1 



3.0 
4.9 
3.0 
3.5 

3.1 
3.4 
2.6 
2.2 

1.7 
1.2 
3.2 
8.2 

4.6 
5.9 
6.6 
4.9 
5.2 



5.6 



3 
3 
12 
5 

4 
5 
6 

4 
3 
\ 

7 

5 
0 
3 
6 
3 



80 



90.7 
91.3 
91.6 
90.4 

87.5 
87.4 
87.7 
85.9 

90.8 
88.7 
88.3 
81.4 

87.4 

85.7 
81.5 
82.3 



1.2 
1.2 
2.1 
1.5 

2.9 
2.6 
2.1 
3.4 

3.7 
0.5 
2.1 
3.1 

4.3 

3.1 
2.8 
6.1 



12 
16 
41 
19 

13 
16 
22 
13 

19 
16 
20 
17 

25 
12 
12 
14 
15 



87.9 
87.8 
87 6 
87.5 

86.6 
86.5 
86.1 
86.3 

85.9 
85.6 
84.8 
82.9 

82.2 
81.3 
80.8 
78.9 
77.1 



3.1 
4.1 
4.2 
4.1 

1.1 

3.2 
3.7 
3.5 

4.0 
4.1 
4.0 
6.3 

7.4 
7.1 
6.6 
5.9 
7.9 



87.7 4.2 302 84.5 6.3 



"Formula for unbiased estimate used. 



*.An unsatisfactory ^{/^ grade was assigned a numerical value of 65 for computation purposes. 



checkpilots was on the Advanced checkride where the means ranged from a low 
of 68.7 (Checkpilot Q) to a high of 89.0 (Checkpilot E)— a difference of more than 
20 points. Similarly, on the Pre -Solo checkride the standard deviations ranged 
from 0.5 to 9.4 (Checkpilots E and M). 

Some of the checkpilots showed a high degree of consistency from one stage 
to another in terms of their relative standing in the group. For example. Check- 
pilot Q assigned low grades. at all three stages of training, while Checkpilots 
A and B assigned high grades for all three stages. 

The data for total checkrides, all stages combined, were subjected to an 
analysis of variance, which indicated that the wide differences in mean grades 
assigned by the checkpilots would not be likely to occur by chance {p< .001). 
Thus, there were statistically reliable differences among these checkpilots in 
the average checkride grades assigned to students. 

The Special checkpilots used in the second phase experiment did not show 
the same degree of variation in grades assigned to students in Classes 63-1 W 
and 63-3. Mean grades assigned under the special conditions established for 
these two classes ranged from* 79.7 to 84.7 and were not significantly different 
from each other. ^ The number of unsatisfactory grades assigned by this group, 
however, was much higher than for any other group of checkpilots discussed 
in this report. Table 6 compares the number of checkrides administered and 
failed in the seven FY 1962 classes listed in Table 1 and the two FY 1963 classes 
reported. The percentage of unsatisfactory checkrides was twice as high in 
the two FY 1963 classes, and all of the unsatisfactory grades in these classes 
were assigned by the Special checkpilots. The failure rate for these Special 
checkrides, therefore, was three times as high as that of the FY 1962 classes. 

These data indicated that there was a significant relationship between the 
number of unsatisfactory checkrides recorded and whether the checkpilot was 
from within or from outside the student's own flight (Chi square = 5.28, p< .05). 
The student whose performance was evaluated by a checkpilot from his own 
training flight was more likely to pass, all else being equal, than was the stu- 
dent who had to demonstrate his proficiency to an unfamiliar or outside evaluator. 

Eight of the 17 checkpilots in the FY 1962 classes had administered at least 
four checkrides for each of the three stages of training. These may be identified 
in Table 5 as Checkpilots C, D, E, G, I, K, L, and M. The mean scores 
assigned by these eight checkpilots for all stages combined ranged from 82.2 
(Checkpilot M) to 87.6 (Checkpilot C). Analysis of variance again indicated 
that these differences are statistically significant (p<.001). 

Table 6 

Checkride Failures In Selected Classes 





Seven 
1962 
Classes 


Ciiisscs 63- IW an( 


\ 63-3 


Item 


Special 
Check- 
ri<lcs 


He pill sir 
Check- 
rides 


All 
Check- 
rifles 


Number of 
Chcckridcs 
Administered 


777 


117 


6.2 


179 


Number Failed 


26 


12 


0 


12 


Percent b'litled 


3.3 


10.3 


0.0 


6.7 



'The alternate Special checkpilot was not included in this analysis. 
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The mean grades assigned by these eight checkpilots differed significantly 
(£<.001) between stages of training. The means for the eight pilots were 83.8, 
85.1, and 88.8 for the Pre-Solo, Advanced, and Cross-Country stages, respec- 
tively. Thus, the grade the average student received was also related to the 
stage of training he was completing. 

While the grading system under which these grades were assigned does 
not define average performance in terms of a specific grade level, it does 
require that all grades be related to the level of student training (5, p. 3). This 
is interpreted to indicate that any grade should have the same implications con- 
cerning relative student proficiency at one stage of training that it has at any 
other stage. Thus, the average grade should be relatively constant for all stages 
of training. Such was not the case in the grades reported here, since the 
"average" student "improved" from a grade of S3. 8 to 88.8 as he progressed 
from one stage of training to another. 

Assignment of students to checkpilots in the Rotary Wing Department was 
not always a random procedure. Occasionally, students having difficulty with 
a particular set of flight skills were assigned to a checkpilot judged especially 
competent to evaluate those skills. Since such assignments would concentrate 
atypical students with certain checkpilots, the results of the analyses of vari- 
ance would be influenced to some extent. It was considered desirable, there- 
fore, to determine whether the bias resulting from this system of assigning 
students to checkpilots was sufficient to account for the significant differences 
among checkpilots in the assignment of cneckride grades. 

Since there were significant differences by stage of training in both instruc- 
tor evaluations and checkrides grades (p<.001), the three stages of training 
were analyzed separately. Therefore, differences between checkpilots could 
not occur as a result of one checkpilot* s having given more checkrides at one 
stage of training than at another. Three separate analyses of covariance were 
performed on the end- of- stage evaluations assigned the students by each of 
the 17 checkpilots identified in Table 5. The covariance technique allowed a 
test of the significance of between-checkpilot differences in checkride grades 
that was independent of the differences in the quality of students as indicated 
by end-of-stage instructor evaluations. 

The covariance analyses showed that there were significant differences, 
other than those attributable to differences in student performance as indicated by 
instructor evaluations, among checkpilots for both the Advanced and the Instru- 
ment Cross-Country stages of training (p< .001). Differences in grades assigned 
by checkpilots at the end of the Pre-Solo stage, however, were no longer sig- 
nificant when allowance was made for differences in instructor evaluations of 
these same students. It should be kept in mind that both the Pre-Solo instruc- 
tor evaluation and the checkride were often administered by the same instructor. 

Thus, in spite of instances in which checkpilots were not assigned.a typical 
cross-section of students for checkrides, significant differences did exist 
among checkpilot grading standards that could not be attributed to inequalities 
in the assignment of students. 

Discussion 

The foregoing analyses of data available from FY 1962 and FY 1963 classes 
of advanced rotary wing training showed that end-of-stage evaluations of student 

*As noted earlier, the most common exception is the Pre-Solo checkride, which may be administered by 
the student^s own instructor. 



performance were significantly affected by (a) the individual standards of 
the checkpilot, (b) whether the checkpilot was a member of the student's own 
training flight, and (c) the stage of training at which flight performance was 
being evaluated. 

The first two factors affecting grade assignment are of particular concern 
for quality control purposes. Whether a student received a Below Average, 
Average, Above Average, or Failing grade on his checkride depended to a con- 
siderable extent on whether he was assigned a "hard" or an "easy" checkpilot. 
If the checkpilot happened to be from another. flight, the student's chance of 
failing the checkride was even greater. 

The influence on checkride grades of the stage of training is of lesser con- 
cern. A constant error of this tjrpe may be expected to have little direct effect 
upon the relative standing of the students within a class. The general lowering 
of Pre-Solo stage checkride grades in the classes involved in this study did not 
appear to have a detrimental effect upon the RotaryWing Department's trainee 
output rate; the Department's attrition rate at the time of these studies was low. 

Findings from this research phase indicated that individual checkpilots did 
have standards and grading practices which differed enough that a student's 
grade could be influenced significantly by chance factors in assignment of stu- 
dents to checkpilots for evaluation. The findings to that effect reported in the 
1956-57 study were, therefore, still valid in the Rotary Wing Department as 
late as FY 1963. While the student's demonstrated flight proficiency undoubt- 
edly influenced the flight evaluation grade hie received, for quality control 
purposes it would be desirable to increase the relationship between grades 
and student flight proficiency and to reduce the differences in standards 
between checkpilots. 

ANALYSIS OF FLIGHT TRAINING GRADE SLIPS 

Problem an d Approach 

The earlier research activities were concerned with the numerical grades 
assigned by checkpilots to represent the overall quality of student flight per- 
formance. The assignment of a numerical value to a student's performance is 
only one of the functions of the Uniform Flight Grading System, the evaluation 
system in use by the Department of Rotary Wing Training, USAAVNS, at the 
time of this study (5). Another objective is to provide information to the Depart 
ment about details of student flight performance, an objective similar to that of 
the inspection and grading functions performed by "quality control" systems in 
other training and production organizations (1). 

Analyses performed in the earlier phases of this study indicated that there 
were limitations on the information that the Uniform Flight Grading System was 
providing about the quality of student performance. The grades assigned by the 
checkpilot "inspectors" were found not to be independent and unbiased evalua- 
tions of trainee quality. Therefore, the effectiveness of any corrective action 
based upon these evaluations was necessarily limited. 

In order to detect deficiencies in the training program and to standardize 
training, detailed information is needed about ways in which student perform- 
ances deviate from the desired standards. Information of this type was provided 
through the Uniform Flight Grading System by means of grade slips being used 
in the Rotary Wing Department at the time of the study. 
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In addition to the numerical^ grade itself, the grade slips provided three 
types of information: (a) identification of proceJures and maneuvers performed 
at or below the desired level of proficiency, (b) evaluations of the student's 
"basic qualities," and (c) descriptions of specific student behaviors which con- 
tributed to, or interfered wit*^, successful performance of certain procedures 
or maneuvers. The information contained in (c) tended to duplicate and to 
elaborate upon that contained in (a). 

The identification of improperly performed maneuvers and the descriptions 
of the student's behaviors which resulted in the improper performance of those 
maneuvers provide information of the type useful for quality control purposes. 
The evaluation of the "basic qualities" by the Department's checkpilots (5) pro- 
vides a suggestion of the causes of low student proficiency— for example, lack 
of motivation. While the "basic quality" concept may contribute to other pur- 
poses of the Uniform Flight Grading System, it lacks the specificity that is a 
necessary characteristic of information required for the effective control of 
the quality of training. 

Procedure 

In order to explore the potential value to a quality control system of the 
descriptive information provided by the grade slips, a number of these grade 
slips were reviewed by research staff members. This review suggested that 
information appropriate for accurate proficiency measurement was being 
recorded, but that the collection of this information depended upon the initia- 
tive of the instructor rather than being set up in the systematic manner nec- 
essary for effective quality control. 

The information was being reported by the checkpilots in the form of narra- 
tive comments on the reverse side of the grade slips. These comments not 
only identified the maneuvers being performed improperly but, in many instances, 
contained specific descriptions of behaviors which caused the student to be 
"marked down'* on the maneuver. This jspecific information could provide data 
for corrective action. An effective quality control system is dependent upon 
specific information of the type represented by many of the comments that were 
being provided by rotary wing checkpilots. 

To determine the amount and kind of narrative information which was avail- 
able, comments made by checkpilots during selected FY 1962 classes were 
reviewed, and a four-category classification was developed. This classification 
provided a means of identifying the level of specificity of each comment,, since 
the usefulness of a comment to a quality control system is proportional to its 
specificity. The four-category classification* follows, with examples of com- 
ments from actual grade slips to illustrate each category: 
Category 1: Suminnry Statement of Performance 
Examples: *'This$ was* a high average ride.*' 
**Average check flight." 
Category 2: Allusion to Specific Disscrepanctes 
Examples: ''Minor discrepancies noted and discussed with student.'' 
''Most of student's work is above average." 

'Under the Uniform Flight Grading System, numerical grades are assigned to checkride evaluations, and 
letter grades are assigned to all other dual flight evaluations, i.e., following each di:nl instructional flight, the 
instructor assigns a letter grade reflecting his evaluation of the student's performance. 

^The reliability of assignment of comments to this classification, estimated by the contingency coeffi- 
cient corrected for u restricted upper limit, was found to be .96 when independent raters were used. 
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Category 3: Specific Discrepancies Noted but Not Related to 
Specific Flight Situation 
Examples: ^RPM Control - Over red line three times. 

""RPM Control ' Not alert for changes in RP.M. 
Weak throttle-pitch coordination." 

Category 4: Specific Disercpancies Noted and Related to Specific 
f'Might Situation 
Examples: Take-off to Hover - Jumped aircraft off ground, 
allowed nose to turn to right, lost RPM down to 
2250. Hovered at 10 feet.'' 
^Airwork - Airspeed dropped to 45k from normal 
cruise (70k) while tuning radio.'' 

Results 

Table 7 contains a tabulation of the comments, categorized according to the 
classification system, made by four of the Special checkpilots' fo^" the 101 
checkrides they administered during the second phase of the research. 

Table 7 

Summary of Checkpilot Comments 
About Student Performance^ 



Chcckpilot 
Number 


Number of 
Chockridcs 


Comment (latcgory 


Total 
Comments 


1 


2 1 3 


4 


I 


28 


I 


4 48 


28 


81 


2 


23 


7 


5 12 


10 


34 


3 


22 


7 


4 13 


24 




4 


28 


17 


2 13 


14 


46 


Totals*^ 


10 1 


32 


15 86 


76 


209 



"Frequencies in Comment Categories 1 and 2 were com- 
bined, and a Chi square of 43.59 (p<.001) was^obtained. 

^Other checkpilots administered 78 checkrides in these 
two classes and made 187 comments, an average of 2.4 com- 
ments per checkride compared with the average of 2.1 comments 
per checkride for the four Special checkpilots. 

Discussion 

The following observations were made concerning the data in Table 7: 

(1) There was no uniform format for the narrative reports of student 
performance. A uniform format would have resulted in a greater concentration 
of comments in a single category. 

(2) The total number of comments made varied significantly among 
checkpilots, even though, as previously reported, there was no significant dif- 
ference among the grades assigned by the checkpilots.* Although Checkpilots 1 
and 4 administered 28 checkrides each, Checkpilot 1 made nearly twice as many 
comments about the students he checked as did Checkpilot 4. 

(3) The usefulness of the information provided, as indicated by the 
categories into which the comments fell, was a function of the checkpilot who , 
administered the flight. For example, specific comments of the type most use- 
ful to a quality control program (Category 4) were made 50 percent of the time 
by Checkpilot 3 and only 22 percent of the time by Checkpilot 2. 

'The Qltcrnatc Special chcckpilot was omitted from this analysis. 
*Scc page 14. 
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(4) Checkpilots did attend to the kind of specific information about 
student performance needed for an effective system of quality control. Although 
this information was not recorded in a systematic manner, the fact that each 
checkpilot noted and recorded specific discrepancies in students' performance 
is significant. It indicates that checkpilots can gather detailed inforiiiation 
about student performance, and that they see the necessity of such information 
for an adequate description. 

The adaptation of information such as was found to be available from the 
grade slips prepared bj^he Rotary Wing Department's checkpilots to a program 
of student performance quality control depends upon the establishment of pro- 
visions for uniform observation and reporting of student behaviors by all check- 
pilots. For quality control purposes, evaluation information should be comparable 
in amount, type, and validity regardless of who provides the information. 

IMPLICATIONS FOR QUALITY CONTROL 

Fljght Training Quality Control 

The establishment of a system for the uniform evaluation of student flight 
performance is no simple matter. Nevertheless, many of the procedures and 
techniques necessary for such a system have been established in actual prac- 
tice. The desired characteristics of such a system include: 

(1) Comprehensive and consistent testing of student flight proficiency. 

(2) Accurate and equitable evaluation of the efficiency of 
training personnel. 

(3) A high degree of uniformity of flight check procedures and 
scoring practices. 

(4) Objective and detailed School standards by which individual students 
or classes can be evaluated. 

The present study indicated that the evaluation data used at the Aviation 
School in 1961-63 had some shortcomings in terms of the broad requirements 
of a quality control program. The results from rotary wing training evaluations 
may be summarized as follows: 

(1) While there was a relationship between the instructor evaluation 
and checkride grades, this relationship was influenced by whether the check- 
pilot was from within or without the student's flight. This raises some question 
as to the validity and reliability of the subjective grade information. 

(2) There were significant and consistent differences among the check- 
pilots in their individual grading practices. 

(3) While the checkpilot comments provided some detailed information 
on student performance, these comments lacked specificity, and uniformity 
from one checkpilot to another, and there was no mechanism for systematically 
using the information. 

The kind of information necessary for quality control (1) was being col- 
lected by the Rotary Wing Department at the time of this study. To be used with 
full effectiveness for quality control purposes, however, this information must 
be gathered systematically and with a high degree of checkpilot standardization. 
It would be part of a quality control program which contains the following 
additional elements: 

(1) A detailed statement of training objectives based on 
job requirements. 

(2) Effecti^re communications concerning performance of students 
on the checkrides. 
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(3) Effective procedures for corrective action where such action is 
indicated by checkride performance. 

(4) Supervisory support. 

The grading practices research in the present study was not directed toward 
determining the presence and effectiveness of any of these four additional quality 
control elements in the Rotary Wing Department at the time of the study. 

As previously indicated, during FY 1966 the Aviation School took steps to 
extend the quality control system already in operation in the primary phase of 
fixed wing training to all phases of fixed wing training. The use of automatic 
data processing of flight grading information for quality control purposes was 
instituted during FY 1967, for rotary wing training as well as fixed wing train- 
ing. The kinds of outputs from this latter application are very similar to those 
provided by the quality control system at the Primary Helicopter School,, but 
go beyond that to include daily grade information. 

Also during FY 1966-67 the Aviation School embarked on a try out at Fort 
Rucker of the PPDR system for checkride purposes in the Rotary Wing Quali- 
fication Course. Technical assistance in this effort was provided by HumRRO 
Division No. 6 (Aviation). New PPDRs, appropriate to the aircraft and training 
content of the "Q" course, v/ere developed and checkpilots were trained in 
their use. As a result of this tryout the Aviation School concluded that the 
PPDR was inappropriate for their requirements, even though many of the 
advantages of objective flight performance recording were recognized. The 
principal areas of difficulty the School found in using PPDRs were related to 
the necessity of having separate and independent groups of personnel who spe- 
cialize in administration of checkrides (this aspect of School organization was 
discussed in the first section of this report). In addition, possible flight safety 
problems were noted by the School, as a result of the considerable mixture of 
aircraft types (fixed and rotary wing), training curricula, and student types. 
Additional exploration toward developing new measures of performance— non- PPDR, 
but yielding detailed and objective results— will therefore be needed in further- 
ing the School's quality control objective. 

In a related activity, personnel in the Department of Primary Fixed Wing 
Training, U.S. Army Aviation School Element, Fort Stewart, Georgia, have 
recently (FY 1967-68) been preparing fixed wing PPDRs for possible tryout. 
HumRRO Division No. 6 has also provided technical assistance for this effort. 
The administrative problems noted in the PPDR tryout at Fort Rucker— that is, 
the requirement for a separate checkpilot section— are not likely to be a matter 
of concern at Fort Stewart, nor should there be as much difficulty resulting 
from mixtures'of aircraft, curricula, and so forth. In both these respects. Fort 
Stewart is more like the Primary Helicopter School at Fort Wolters than the 
Aviation School at Fort Rucker. 

General Quality Control Implications 



While the occasion initiating the present study was USAAVNS interest in 
improving quality control practices and procedures at that school, for HumRRO 
a corollary objective was to develop implications for quality control in general. 
Study and analysis were focused on aspects of the proficiency data not only with 
reference to flight evaluation, but also for possible implications with regard to 
proficiency data characteristics applicable in any formal quality control system 
While such a system is comprised of more than proficiency data, this informa- 
tion is the fundamental ingredient for quality control. 
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The results of this study suggest a number of general implications about 
ways in which proficiency data may be made maximally useful for quality control: 

(1) Two distinct functions of proficiency data should be recognized in 
training quality control— description of performance and evaluation of perform- 
ance. It is especially critical for quality control purposes that the data be 
descriptive of the actual performance if records of the performances of numerous 
students are to be pooled to provide indications of how the overall training 
system is functioning and where and what corrections should be instituted. 

(2) The basic data on performance proficiency are most valuable when 
they are objective— that is, when the data are close to being clear, unambiguous 
observations, free of personal judgment. With objective data, observations do 
not depend upon who is observing, and there can be assurance that the profi- 
ciency data' are a dependable measure of performance rather than a reflection 
of personal or other factors in the testing situation. 

(3) Proficiency testing should be done with st andardized subject matter 
and under carefully standardized conditions, so the evaluations will be demon- 
strably comparable in content and administration, regardless of who is being 
tested or who is doing the testing. 

(4) To attain maximum objectivity, the collection of data on proficiency 
needs to be clearly separated from the process of evaluating the performance 
reflected in the data. 

(5) Criteria and standards for evaluation need to be developed in : .ich 

a way that they are uniform for all evaluators, for all students, and for all stages 
of training, 

(6) Diagnostic information regarding the effects of training is needed, 
specifically and in detail, if the needs for modifying training are to be clearly 
identified. Detailed instructions to evaluators and a uniform format for report- 
ing on a student's performance can help make diagnostic information most usable. 

(7) Ideally, for quality control purposes, proficiency testing and eval- 
uation should be performed by an administering unit and personnel who are 
independent of the training function. At a minimum, the evaluators ought to be 
separate from the personnel directly concerned with training the particular stu- 
dents being evaluated, 

(8) In order to insure objectivity and standardization in performance 
observation and evaluation, quality control personnel need special preparation 
in the concepts and techniques of observation, measurement, and evaluation. 

Implications for Future Research 

Basic concepts, techniques, and procedures for quality control systems 
have been in existence for a long time, especially those having to do with indus- 
trial production of material goods, so "ideal" conditions for quality control are 
quite well known. It is not always possible, however, to achieve ideal conditions 
since a quality control system— whether it be industrial or training— must fit 
into the resources that can be made available by the organization. A fruitful 
avenue for further study is in determining how to consider and evaluate the 
trade-offs between the ideal and the practical— that is, what kind of lose would 
be incurred by what kind of deviation from the ideal? In considering establishing 
an ideal feature in quality control, knowing the loss without it— compared to the 
probable cost of adding it— can provide a basis for establishing the best cost- 
benefit balance, and therefore, for practical solutions to quality control that 
are realistic in terms of the facilities and needs of the school or training center. 
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There are a number of other promising directions for further study of 
flight training quality control, especially to consider situations where adminis- 
trative or other factors preclude the kind of specialized evaluation unit available 
to USAPHS. Essentially, they are approaches to providing detailed proficiency 
measures that will meet quality control needs without specialized personnel 
trained in measurement and evaluation to the extent needed in PPDR-type systems. 

One direction for study is development of different types of objective meas- 
ures of performance. In HumRRO Work Sub-Unit ECHO III, for example, a 
time-lapse motion picture camera has been used to record instrument panel 
readings in flight. If this approach proves to be practical and effective in terms 
of the cost and the quality of information developed, detailed objective perform- 
ance information for quality control purposes could be collected automatically 
for analysis after a flight. Such analysis might be done by clerical rather than 
checkpilot personnel, and could be completely independent of the evaluation 
fuiiCtion. While use of a camera could minimize the detailed and standardized 
observations required of the checkpilot, it would make it especially critical that 
all students perform a standard set of test maneuvers under standard conditions. 

A contrast was drawn earlier between the PPDR— a relatively objective 
description of performance—and the Uniform Flight Grading System at USAAVNS— 
a relatively subjective evaluation of performance. One possibility for a study 
is to consider combinations of these approaches. Consideration of various 
flight proficiency studies suggests that a subjective-objective approach to flight 
proficiency measurement may be promising; it might be feasible to develop 
some joint system that would be practical and effective and yet require less 
specialized training for checkpilots. 

Still another direction for study concerns the base for comparison used 
in flight proficiency evaluation. Generally— assuming a basic minimum pro- 
ficiency above a failing level— flight evaluation tends to be based on comparing 
performance among students in terms of above or below "average" proficiency 
for a given level of training. An alternate approach would be to evaluate a stu- 
dent's performance using an anchored scale that would allow comparison of his 
'Capabilities with those of an expert pilot. It might not be desirable or economi- 
cal to bring student pilots to an expert's level, but using expert performance 
as an anchor point for scaling student performance could be a way of empha- 
sizing the perspective of operational capability as the goal for training. Such s 
change in scaling would be likely to affect what portions of training are iden- 
tified for improvement, on a basis of operational needs rather than inter- 
student comparisons. 
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Appendix A 



MEAN DAILY GRADES, INSTRUCTOR EVALUATIONS, 
AND CHECKRIDE GRADES FOR EACH STAGE OF TRAINING 



Table A- 1 



Means and Standard Deviations of Score: 
for Selected FY 1961 Advanced Rotary Wing Training Classes 



Training Stage 


Varialilc 


Class Numlicr 


All 

Ciasrvc.s 
(A- =1.57) 


61-1 

(iV=36) 


61-2 

(iY=30) 


61-3 

(A'=28) 


61-4 

(iV=29) 


61-5 
(A = 19) 


61-6 

(A' =^15) 


Prc-Solo 


Daily Gradc^ 


















Mean 


3.0 


2.9 


2.9 


3.0 


2.9 


3,0 


2.9 




SI) 


.21 


.24 


.20 


.21 


.29 


.17 


.22 




Instructor Evaluations 


















Mean 


85,8 


83.6 


84.9 


84.9 


84.4 


83.9 


84.7 




SD 


3.4 


4.9 


4.8 


4.7 


4.0 


5.0 


4.5 




Chcckridc Grades 


















Mean 


84.5 


79.2 


83.1 


83.3 


at.2 


83.5 


82.9 




SD 


4.6 


7.1 


6.3 


6.0 


6.5 


8.3 


6.6 


Advanced 


Daily Grade* 


















Mean 


3.1 


3.0 


3.0 


3.0 


3.1 


3.1 


3.0 




SD 


.22 


.19 


.37 


.24 


.24 


.11 


.25 




Instructor Evaluations 


















Mean 


86.8 


85.3 


85.6 


85.3 


87.7 


86.8 


86.1 




SD 


3.4 


4.7 


5.9 


4.9 


3.1 


4.1 


4.6 




Chcckridc Grades 


















Mcnn 


82.6 


83.9 


84.0 


83.8 


85.5 


85.3 


83.9 




SD 


7.2 


5.0 


5.7 


6.0 


4.8 


4.1 


5.9 



'''Daily grailos uorc scaled on the basis of: Kxccllcnt (A) 4; Good (B) = 3; Low to Fair (C) = 2; and 
Unsatisfactory (U) = 1- 
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Table A-2 



Means and Standard Deviations of Scores 
for Selected FY 1962 Advanced Rotary Wing Training Classes 



training Stage 


\ nriabic 






Cla^s Numh 








AM 
(A = 259) 


62-Itt 

(A' ^38) 


62-2\V 
(A' = 42) 


-62-3 
(A- ^35) 


62-5 
(A' = 45) 




62-6 
(A'= 28) 




62-7 

(A'= 27) 


62-8 
(A 43) 


Prc-Solo 


Daily Grade'* 




















Mean 


2.9 


3.0 


2.9 


3.0 


2.9 


3.0 


2.9 


2.9 










.36 


.26 


.JO 




.J') 






Instructor bvaluations 




















.Mean 




85.0 


85.2 


85.3 


84.2 


o\. 1 


85. 1 


85.0 






4.1) 


1 0 


4.6 


.5.0 


•I "7 
O. I 


O.J 


4.0 


f.f) 




CJieckride Grades 




















.Mean 


OO 1 


-oU. 1 


83.8 


84.4 


83.4 


82.8 


O.J.I 


82.8 






I .u 


U 1 


6.5 


6.0 


O.J 


I .u 


f). 1 


o.y 


Advanced 


Daily Grade ^ 




















Mean 


2.9 


2.9 


3.0 


3.0 


3.0 


3.0 


2.9 


3.0 




SD 


.33 


.27 


.18 


.29 


.16 


.26 


.25 


.•26 




Instructor [^valuations 




















Mean 


85.8 


85.0 


87.5 


87.2 


86.6 


85.7 


85.2 


86.1 




SD 


4.8 


5.5 


3.9 


4.3 


3.1 


3.7 


4.7 


4.6 




Checkride Grades 




















Mean 


82.8 


81.6 


83.3 


84.1 


86.3 


83.8 


82.8 


83.4 




SI) 


6.9 


7.4 


7.0 


5.1 


4.1 


6.1 


5.5 


6.3 


Instrument 


Daily Grade" 


















Cross-Country 


Mean 


3.3 


3.2 


3.0 


3.4 


3.3 


3.2 


3.2 


3.2 




SD 


.31 


.31 


.38 


.31 


.29 


.26 


.35 


.33 




Instructor Evaluations 




















Mean 


88.8 


86.8 


88.0 


89.0 


88.1 


87.8 


86.8 


87.8 




SD 


3.6 


3.3 


4.9 


3.6 


3.9 


3.2 


4.0 


4.0 




Checkride Grades 




















Mean 


87.6 


85.9 


87.6 


87.2 


89.6 


86.6 


85.9 


87.0 




SD 


4.1 


4.7 


4.7 


6,2 


2.7 


6.0 


5.8 


5.2 



^Dally grades were scaled on the basis of : Kxcelleitt (A) ~ 4; Good (H) ^ 3; Low to Fair (C) ~ 2; and 
Unsatisfactory (U) " 1. 
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OlR CF INSTR ARMCR SCH FT KNCx 

CCPOT ARNV Chen corps SCh ft mCCLELLAn ATTN EOUC AOV 
ARPY INF SCh FT BENNlNC ATTN EDUC ACV 
ISA INF SCH ATTN OlR CF iNSTR FT BENNlNC 

CC»'OT ARKV TRANS SCH FT EUSTIS ATTN ECUC AOV 

CCPOT US ARMY SOUTHEASTERN SIC SCh ATTNt EOUC AtVlSOR FT COROON 

ASST COPOT ARMV AIR CEF SCH FT BLISS ATTN CLASSF TECh LIB 

CC ARPV ARTY ♦ MSL CTR FT SILL ATTN AVN OFFft 

CCPOT ARNEO FORCES STAFF COLL NORFOLK 

CCPOT JUOCE AOVOCATE GENERALS SCH U OF VA 

EClC CONSLT ARKV MIL IT POLICE SCH FT CORDON 

CCNOT ARKV ENCNR SCH FT BELVClft ATTN AlBBES'Sv 

CCPOT ARftV AVN SCh FT RUCKER ATTN EOUC AOV 

CCPOT ARMV PRIMY HEL SCH FT MOLTERS 

SPEC WARFARE SCH FT BRACC ATTN LIB 

ISA SPEC MARFARE SCH ATT: COUNT ER INSURCENCY OEP T FT BRACC 

SECV US ARMY ftSL t iniNlTlONS CTR C SCH REDSTONE AllSNL 

HC ABERDEEN PC ATTN TECh LIB 

CO'OT US ARHY INTEL SCH FT hCLABlRC 

OlR 8RC0 ♦ BN OPNS CEPT USAIS FT BENNlNC 

OlR CCfM ELEC USAIS FT BENNlNC 

OIR CCPPANV "^ACTICS CEPT USAIS FT BENNlNC 

CC US ARKV SICNAL CTR * SCh ATTN S 1 ICvL- 3IC0BET 11) 

SECv CF ARMY 

OCS-PERS OA ATTN CHF C*S ClV 

OlR CF PERS STUDIES * RES GCCSPER CA ATTN 8C MALLACE L CLEMENT 

CC FCREICN set * TECH CTR NUN BLCC 

ACS FCrt FORCE OEVEL CA ATT» ChF TNC DlV 

HC ARPV f«AT COftO R*C CRCTE ATTN AMCRD-RC 

CC ARPV PED R*D CCKC ATTN BEhAV SCI RES BR 

IS ARPV BEHAVIORAL SCI RES LAB MASht CC. ATTNt CRO-AR 

CFC PERS KCT DEV CFC ATTN MOS SEC INEW EOUlPI OPOmO 

Aill'V PROVOST HARShAL CEN 

CFC RESERVE COftPCN CA 

ADPlN DDC ATTNt TCA IHEALY) CAMERON STA ALEX.t VA. 22314 

CC US ARMY MEO RES LAB FT KNCX 

CC ARPV ELECT COPO FT NCNHOUTH ATTN ANSEL C6 

ChF CF RtO OA ATTN CHF TECh * INCSTR LIAISON OFC 

PERS * TNC OlV OROhC CFC OF ChF CF ORG CA 

CC ARPV HEO R*0 CCMO ATTN NECCH-SR 

I S ARMY BEHAVIORAL SCl RES LAB MASh* O.C. ATTN CRO-aIC 
CCPOT ARMY CBT SURVEIL SCH FT hUAChUCA ATTN ATSUR S3 
CC ARPV AIR OEF UfC ENT AFB 

CC US AR»(V f<AT CCKC MASh CC. ATTNt AMCPT-CM ROBT OETlENNE 

PRES ARHV MAlNT BG FT KNOX 

PRES ARMV AVN TEST BC FT RUCkER 

OPTY PRES ARKY f<AT CCMC BC ABERCEEN PC 

tS ARPV ARCTIC TEST CTR R C C OFFICE SEATTLE 

CC 20 ARPOREO OlV FT hCCO ATTN ClV AVN OFCR 

CC 4TH AROOREO CIV APC 09)26 NY 

CC 20 ARf*ORED CAV RECT APO 09696 NV 

CC )0 ARPOREO CAV RECT APO 090)4 NV 

CC UTH ARMGREO CAV RECT APO 09926 NV 

CC 20 BN )4TH ARftCR FT IRMlN 

CALIF NC 40TH ARHOREC GiV LOS ANCELES ATTN AC OF SC3 

iiJti COnO HC CIV ARPV NC JACKSONVILLE FLA 

CC DOTH AVfi BN NJ AIR NC ELIZABETH 

CC HC 27TH /RPOREO CIV NY A|R HC SYRACUSE 

TEXAS NC 49TH ARPOREC GlV CaLLAS 



CC 1ST Ihf OlV FT RILEV ATTN C3 

CO )R0 INF OlV ATTN C3 NY 

CC 4TH I^F OlV FT LEmIS ATTN C3 

CC 7TH INF OlV San FRAN ATTN C2 

CC 8TH iNF OlV ATTN C2 APO 09111 Ht 

CC )1H INF OlV (PECh) FT CARSCN 

CC ?4TH UF OlV ATTN C3 APO 09112 NV 

CC 620 ABN INF GlV FT BRACC ATlN G3 

CC 197TH INF 8RCC FT BENNlNC ATTN S3 

CC 2»TH INF OlV APO 9622$ NV 

CC BN 39TH INF APC 09029 NV 

CC 2N0 BN 19TH INF NV ATTN S 3 

CC 1ST BN (KECh) »2C 1^F 1ST ARMOREC ClV lOLO IRONSIOCSI Fl mOOO 
4lH BN (PECH) 54TH iNF FT KNCx 

CC ARPV PARTIC CP NAV TNC CEVlCE CTR PT MASHINCTON ATTN CODE OlA 
CHF A'.OIC VISUAL APPLICAT OFC ARMY PICTORIAL OlV OFC OF ChF SIC OFCft 
CC PILIT OlST CF mAShInCTCN 
^IR ARPV LIB PENTACCN 
CHF CF PILIT HIST CA ATTN CEN R£F 0R 
820 ABN OlV FT BRACC 
CC 24TH ARTV CP (AO) COVENTRY 
CC 31ST ARTV BRCC AIR CEF OAKCALE PENNA 
2BTH ARTV CP AlR CEF SCLFRlCCE AFB 
920 ARTV BRCO AIR CEF hlChLANCS AFS 
HC 49TH ARTY BtCC AIR CEF ARtlNCTGN hTS ILL 
CC ICIST ABN OlV FT CAhPBELL 
CC IST CAV OlV APO 96490 SAN FRAN 

tS ARPV TRCPIC TEST CTR PC CRAMER 9*2 ATTN BEHAV SCIENTIST FT CLAYTON 

CInC tS ATLANTIC FLT COCE 312A NORFOLK ATTN L TC OOTV 

COR TNC CONMANO US PACIFIC FLT SAN CIEGO 

CHF BIR CF MEG * SURC CN ATTN CGCE 913 

OlR PERS RES OlV BUR CF NAV PERS 

TECH LIB BUR CF ShIPS CCCE 210L NAVY CEpT 

CC * OlR NAV TNC CEVlCE rTR CRLANCO ATTN TECh LIB 

CC FLT ANTl'AlR WARFARE TNC SAN CIECO 

CO NLCLEAR WEAPONS TNC CTR PACIFIC U S NAV AIR STA SAN DIECO 
CC NAV AIR OEVEL CTR JChNSVlLLE PENNA ATTN NAOC LIB 

HlPAN FACTORS OEPT CCPM PSvChOL OIV NAV TNC DEVICE CTR PT nAShINCTON 

CL FLEET TNC CTR U S NAV STA SaN CIECO 

CC FLT ANTl'SUB WARFARE SCH SAN OlECO 

CHF CF NAV RES ATTN SPEC ASST FOR R C 0 

CHF CF NAV RES ATTN hEAC PERS * TNC BR CODE 4)8 

OlR IS NAV RES LAB ATTN CCOE )120 

CC CFF OF NAV HES BR OFFICE BOX 39 FPO 09510 NV 

CHF CF NAV AIR TNC TnC RES CEPT NAV AIR STA PENSaCOLA 

CC NAV SCH OF AVN PEC NAV AVN MEO CTR PENSACOLA 

CC PED FLO RES LAB CaPP LEJEUNE 

COR NAV PSL CTR POINT PUGU CALIF ATTU TECH LIB CODE 3022 
OlR AEROSPACE CREW ECUlP LAB NAV AIR ENCNR CTR PA 
CIC NAV PERS RES ACTVV SAN CIECO 

COR NAV mSL CTR PClNT MUCU CALIF ATTN hUMikN ENCNR OlV COOE N'335 

CCPOT MARINE CORPS HC MARINE CORPS ATTN COOE AO-IB 

HC PARlNE CORPS ATTN AX 

Ollt PARlNE CORPS iNST ATTN EVAL UNIT 

CHF CF NAV AIR TECH TnC NAV AIR STA MEMPHIS 

OlR CPS EVAL CRP OFF OF ChF OF NAV OPS 0P03EC 

CCPOT PTP COAST CUARC hC 

CHF SCI OlV ORCTE SCI * TECH GCS R«0 »<Q AIR FORCE AFRSTA 

CHF CF PERS RES BR CRCTE CF CIVILIAN PERS OCS-PERS HO AIR FORCE 

CHF EVAL BRtAFPOPCE) CAREER OEVEL OlV CRCTE OF PERS PLAN HQ AlR FORCE 

FAA ChF INFO RETRIEVAL BR WASH CC. 

FED AVN ACV MEO LIB hC-640 

HC AFSC SC8B ANGREWS AF8 

HC BALLISTICS SvS GlV PERS SUBSvS BR BSOSP NORTON AFB 

6S70TH AERO f^EO RES LAB MRPT WR IChT-P AT TERSON AFB 

AIR PCVEPENT OESICNATCR APRh BROOrS AFB 

COR ElEC SYS OlV LC hANSCCM FLO ATTN ESTl 

OlR AlR U LIB MAXWELL AFB ATTN AUL3T-63-2)3 

AIR FCRCE SCh CF AERCSPACE MEC BrOOkS AFd ATTN AEROmEO LIB 

CCPOR ARCTIC AERCPEC LAB APO 9B731 SEATTLE 

6570TH PERS RES LAB PRA-4 AEROSPACE MEO OlV LACKLANO AFB 

PSvChCBICLCGY PRCC NATL SCI FCUNC 

OlR OlV OF OATA SOURCES C STANO NATl CTR FOR EOUC STATISTICS 

CFC CF EOUC 
OlR NATL SECUR ACv FT CEO C MEaCE ATTN TOL 
CENTRAL INTEL ACv ATTN CCR/CO STANGAO DISTRIBUTION 
SvS EVAL OIV RES DIRECTORATE COO-OCC PENTAGON 
OEPT CF STATE BUR CF INTEL * RES EXTERNAL RES STAFF 
SCI UFC ExCH WASHINCTON 

CHF RECL TNC 8R INC ClV FEO AVN ACV ATTN PT 38 

BlR CF RES t ENCR US POST OFC OEPT AlTN ChF HUMAN FACTORS BR 

EClC PEOIA 8R OE OEPT OF HEW ATTN T C CLEMENS 

CFC CF INTERNATl TNC PLANNINC C EVAL BR AlO WASH OC 

SvS GEVeL CCRP SANTA PGNICa ATTN LlB 

OINLAP * ASSOC InC CARIEN ATTN LIB 

RESEARCH ANALYSIS CCRP PCLEAN VA 22101 

OlR RANO CORP SANTA PCNICA ATTN LIB 

I CF SO CALIF ILEC PERS RES CP 

CCLUPBIA U ELEC RES LABS ATTN TECh EGlTOR 

PITRE CORP BEOFORO PASS ATTN LIB 

U CF PCh LEARNINC R«C CTR ATTN CIR 

HlPAN SCI RES tUC NCRFOLK 

HlPAN SCI RES iNC PCLEAN VA 

TECh Info CTR ENCNR CATA SERV N APER AVN INC COLUMBUS 0 
CHRYSLER CORP MSL ClV DETROIT ATTN TECh iNFO CTR 
EOlC * TNC CONSULTANTS LOS ANCELES ATTN PRINCIPAL SCI 
CEN OVNAMlCS POPCNA CaLTF ATTN LIB 

AVN SAFEtY ENCR C RES OlV OF FLIChT SAFETY FOUND iNC PHDENlX 

PARClARDT CCRP POPCNA CALIF ATTN CEPT 580 

CTIS ELEVATOR CO OlV ATTN LIB STAMFORD CONN 

CHF PERS SUBSvS AIRPLANE DlV MS 74-90 RENTON WASH 

IHICKCL CHEf< CORP HUPETRICS ClV LOS ANCELES ATTN Ll6N 

UST FOR OEF ANLS RES * ENCNR SUPPORT DlV WAShINCTON 

HIChES aircraft CCPPANV CULVER CITY CALIF 

DlR CTR FOR RES CN LEARNINC * TEAChINC U OF MICH 

EOITCR TNG RES ABSTR APER SCO OF TNC ClRS U OF TENN 

HlPAN FACTCRS SECT R*D CEN DYNAMICS ELECTRIC BOAT CROTON 

BRITISH E^tBSY BRlTlSh CEF RES STAFF WAShlNCTON 

CANADIAN JCINT STAFF CFC OF CEF RES MEMBER MAShINCTON 

CANADIAN ARPY STAFF WAShlNCTON ATTN CS02 TNC 



CERfAN lUISCN CFCR kWX AVN TEST 60 FT RUCKER 

AC) FCR IftTEl FCIIE1CI». tUISCN OFCR TO NORMEC Nil IT ATTACHE 

ARfV ATTACHE ROYAl SNiClSh E>'eSV MAShINCTON 

NATl INST FCR AlCCHCl RES OSIC 

OEF RES fEO lAB CnTARIQ 

FRENCH tlAlSCK CFCR AR»*V AVN TEST 60 FT RUCKER 
BRITISH IIAISCN CFCR kKfX AVN T£ST 60 FT RUCKER 
CFC CF AIR ATTACHE AUSTRALIAN ^NBSV ATTN: T.A. NAVCN 

kiASH, O.C. 

AtSTRAlIAK EftBSV CFC CF MIIIT ATTAChE WASHINGTON 

>'ElkMNCER FCUNDATICK TCPEKA 

AKER INST FCR RES SIlV(R SPRING 

A^ER !nST I^CR res POh ATTN IIBN 

OIR PRI>'ATE lAB UKIV CF M IS NAOISON 

fATRIX CCRP ARllKCTCN ATTN TE'':^ I IBN 

A>iER TEl^TEl CO KV 

OR CECRCE T HAUTV Ch>'K CEPT OF PSvOlU OF CEl 
CE^ ELECTRIC CO SANTA SARfiARA ATTN tlB 
VITRC lABS SILVER SPRING ATTN L IBN 
HEAO CEPT CF PSvChCL UKIV OF SC COLUMBIA 
t CF GECrGU CEPT CF PSvChOL 
APER INST FCR RES PALC ALTO CALiF 
PICh state U CCLL CF sec SCI 
K PEX STATE U 

aCttLAKO ♦ CC HAOCC^FIELC NJ ATTN PRES 

NCRTRCNICS OIV CF NCRThROP CCRP ANAhEiN CALIF 

CHIC STATE U SCH CF AVN 

AIRCRAFT ARKACEKTS INC COCKEvSVILLE NC 

TtFTS U HUNAN ENGNR UFC ♦ ANLS PROJ 

HtPAK FACTCRS RES GP WASH U ST LOUIS 

APER PSvCHCL ASSCC WAShINGTCN ATTN PSvCHOL ABSTR 

KC ILL U HEAO OEPT OF PSvCHCL 

OR A. V. FENO STAKFCRC RES INST FT 0 

BELL TEL LAS'; INC TECh INFO LIB WI-IPPANV LAB NJ ATTN TECH REPORTS LIBN 

REPUBLIC Avh CORP FARPlKGCALE LONG ISL ATTN SUf>ERV ENGNR LIB 

LIFE SCI INC FT WORTH ATTN PRES 

AP(R BEHAV SCI CALIF 

CCLL CF WH ♦ liARV SCh CF ECUC 

SC IlLlNCIS U OEPT CF PSvChCL 

IiASj PILITARV SVS CIV BETkESCA NC 

RCA BURLINGTON HASS ATTN: WN R BUSH 

NCRTHhESTERN U OEPT CF INCSTR ENGNR 

HCNEVhELL ORO ST A KAlL ST A 906 N INN 

NV STATE EOUC OEPT ABSTRACT EDITOR AVCR 

AIRCSPACE SAFETV ClV U CF SOUTHERN CALIF LA 

PR BRANDON B SNITH RES ASSOC U OF HtNN 



CTR FOR THE ACVANCCC STUCV CF ECUC 

chf processing civ CUK( U Lie 

t CF CALIF GEN LIB CCfU CEPT • 
FlCRlCA STATE U LIB GIFTS ♦ (XCh 
HAftVARO U PSvCHCL LACS LIB 
t OF ILL LIB SER CEPT 

t cf kansas lib periccical cept 

t cf kebraska libs acc cept 

Chic state u libs gift ♦ exCk oiv 

P(KKA STATE U PATTEE LIB COCU CESK 

PtROLE U LIBS PERICCICALS ChECKiNC 

STAFFORD U Lies OCCU LIB 

LIBK U CF TEXAS 

SVRACtSE U LIB SER CIV 

t CF ^IlkKESCTA LIB 

STATE U CF ICWA LIBS SER ACC 

KC CAROLINA STATE CCLL OH HlLL LIB 

fiCSTOK U LIBS ACC ClV 

t CF MOH LIBS SER CIV 

BRCh^ U LIB 

CCLU^BIA U LIBS COCU ACC 

DiR JOINT U LIBS N*Sl-VILLE 

OIR t LIB GEC WASHINGTON U 

LIB CF OO^GRtSS OhF CF EXOH * GIFT 

t CF PGH OCCU LlBN 

CATHOLIC U LIB ECUC C PSvOHOL Lifi 

t CF KV PARGARET I KiKG LIB 

SC ILL U ATTN LIBN SER OEPT 

KAKSAS STATE U FARRELL LIB 

BRIGhAP VCUKC U lib SCR SECT 

t OF LOUISVILLE LIB BELKNAP CAMPUS 

P E EASTER SCH CF AVN ChlC 

R E FLEXMN OlR OF ACv TNG 

I* B BOhEV CKR NTCO FLA 

H kERBER life SOI RSCH GOOCV(AR 

E HALL PELPAR FALLS CHJROh VA 

0 E PEVER TNG RSCH ClV fEHAV 

C B POTTER OIR AVh SAFETV ClV 

CFC OF SURGEON GEN ATTN AVN 

tSA TRANS RSCH OOPC ATTN F KOCURT 

R A PONTV CORNELL AEHONAuTlOAL LAB 

K B CEGREENE UNIV OF S CALIF 

LEARMNG RESOURCES CEN UNIV OF 

L h CtRETON SCH CF ECUC UNlV 

C REIC SPRINT HUPAN FACTORS 

J A GRCSSLIChT OEPT CF PSvOh 



ACmIN ATTN iONE PIERRON U OF OREC 



ClV 
WASH 00 



STATE UHlV COLUMBUS 

REQUIREMENTS LINK PRECISION 00 NV 

AFROSPACE CO OHIO 

SOI LAB WRIGHT-PATTERSON AFB 
UNIV OF S CALIF 
SEC OA WASH 0 0 
FT EUSTIS VA 
BUFFALO NV 

TCNN KNOXVlLLE 
OF TENN KNOXVULE 
MARTIN CO FLA 
FLA STATE UNIV 
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